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Modeling  Item  Responses  When  Different  Subjects 
Employ  Different  Solution  Strategies 


Abstract 


A  model  is  presented  for  item  responses  when  different 
examinees  employ  different  strategies  to  arrive  at  their  answers, 
and  when  only  those  answers,  not  choice  of  strategy  or  subtask 
results,  can  be  observed.  Using  substantive  theory  to 
differentiate  the  likelihoods  of  response  vectors  under  a  fixed 


set  of  solution  strategies,  we  model  responses  in  terms  of  item 
parameters  associated  with  each  strategy,  proportions  of  the 
population  employing  each,  and  the  distributions  of  examinee 
parameters  within  each.  Posterior  distributions  can  then  be 
obtained  for  each  examinee,  giving  the  probabilities  that  they 
employed  each  of  the  strategies  and  their  proficiency  under  each. 
The  ideas  are  illustrated  with  a  conceptual  example  about  respons 
strategies  for  spatial  rotation  items,  and  a  numerical  example 
resolving  a  population  of  examinees  into  subpopulations  of  valid 
responders  and  random  guessers. 
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Introduction 

The  standard  models  of  item  response  theory  (IRT) ,  such  as 
the  1-,  2-,  and  3 -parameter  normal  and  logistic  models, 
characterize  examinees  in  terms  of  their  propensities  to  make 
correct  responses.  Consequently,  examinee  parameter  estimates 
are  strongly  related  to  simple  percent-correct  scores  (adjusted 
for  the  average  item  difficulties,  if  not  all  examinees  have  been 
presented  the  same  items).  Item  parameters  characterize  the 
regression  of  a  correct  response  on  this  overall  propensity  toward 
correctness . 

These  models  lend  themselves  well  to  tests  in  which  all 
examinees  employ  the  same  strategy  to  solve  the  items. 

Comparisons  among  estimates  of  examinees'  ability  parameters  are 
meaningful  comparisons  of  their  degrees  of  success  in  implementing 
the  strategy.  Item  parameters  reflect  the  number  or  complexity  of 
the  operations  needed  to  solve  a  given  item  (Fischer,  1973). 

The  same  models  can  prove  less  satisfactory  when  different 
examinees  employ  different  strategies.  The  validity  of  using 
scores  that  convey  little  more  than  percent-correct  to  compare 
examinees  who  have  used  different  strategies  must  first  be  called 
into  question.  And  item  parameters  keyed  only  to  a  generalized 
propensity  toward  correctness  will  not  reveal  how  a  particular 
kind  of  item  might  be  easy  for  examinees  who  follow  one  line  of 
attack,  but  difficult  for  those  who  follow  another. 
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Extensions  of  IRT  to  multiple  strategies  have  several 
potential  uses.  In  psychology,  such  a  model  would  provide  a 
rigorous  analytic  framework  for  testing  alternative  theories 
about  cognitive  processing  (e.g..  Carter,  Pazak,  and  Kail,  1983). 
In  education,  estimates  of  how  students  solve  problems  could  be 
more  valuable  than  how  many  they  solve,  for  the  purposes  of 
diagnosis,  remediation,  and  curriculum  revision  (Messick,  1984). 
And  even  when  a  standard  IRT  model  would  provide  reasonable 
summaries  and  meaningful  comparisons  for  most  examinees,  an 
extended  model  allowing  for  departures  along  predetermined  lines 
(e.g.,  malingering)  would  reduce  estimation  biases  for  the 
parameters  in  the  standard  model. 

In  contrast  to  standard  IRT  models,  and,  for  that  matter,  to 
the  "true  score"  models  of  classical  test  theory,  a  model  that 
accommodates  alternative  strategies  must  begin  with  explicit 
statements  about  the  processes  by  which  examinees  arrive  at  their 
answers.  For  example,  items  may  be  characterized  in  terms  of  the 
nature,  number,  and  complexity  of  the  operations  required  for 
their  solution  under  each  strategy  that  is  posited. 

The  recent  psychometric  literature  contains  a  few 
implementations  of  these  ideas.  Tatsuoka  (1983)  has  studied 
performance  on  mathematics  items  in  terms  of  the  application  of 
correct  and  incorrect  rules,  locating  response  vectors  in  a  two- 
dimensional  space  based  on  an  ability  parameter  from  a  standard 
IRT  model  and  an  index  of  lack  of  fit  from  that  model.  Paulson 
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(1985),  analyzing  similar  data  but  with  fewer  rules,  uses  latent 
class  models  to  relate  the  probability  of  correct  responses  on  an 
item  to  the  features  it  exhibits  and  the  rules  that  examinees 
might  be  following.  Yamamoto  (1987)  combines  aspects  of  both  of 
these  models,  positing  subpopulations  of  IRT  respondents  and  of 
non-scalable  respondents  associated  with  particular  expected 
response  patterns.  Samejima's  (1983)  and  Embretson's  (1985) 
models  for  alternative  strategies  are  expressed  in  terms  of 
subtasks  whose  results  are  observed,  in  addition  to  the  overall 
correctness  or  incorrectness  of  the  item. 

The  present  paper  describes  a  family  of  multiple-strategy 
IRT  models  that  apply  when  each  examinee  belongs  to  one  of  a 
number  of  exhaustive  and  mutually-exclusive  classes  that 
correspond  to  an  item-solving  strategy,  and  the  responses  from 
all  examinees  in  a  given  class  are  in  accordance  with  a  standard 
IRT  model.  It  is  further  assumed  that  for  each  item,  its 
parameters  under  the  IRT  model  for  each  strategy  class  can  be 
related  to  known  features  of  the  item  through  psychological  or 
pedagogical  theory. 

The  next  section  of  the  paper  gives  a  general  description  of 
the  model.  It  is  followed  by  a  conceptual  example  that 
illustrates  the  key  ideas.  A  two-stage  estimation  procedure  is 
then  presented.  The  first  stage  estimates  structural  parameters: 
basic  parameters  for  test  items,  examinee  population 
distributions,  and  proportions  of  examinees  following  each 


Different  Strategies 

4 

strategy.  The  second  stage  estimates  posterior  distributions  for 
individual  examinees:  the  probability  that  they  belong  to  each 
strategy  class  and  the  conditional  distribution  of  their  ability 
for  each  class.  A  numerical  example  resolves  examinees  into 
classes  of  valid  responders  and  random  guessers.  The  final 
section  discusses  some  implications  of  the  approach  for 
educational  and  psychological  testing. 

The  Response  Model 

This  section  lays  out  the  basic  structure  for  a  mixture  of 
constrained  item  response  models.  Discussion  will  be  limited  to 
dichotomous  items  for  notational  convenience,  but  the  extensions 
to  polytomous  and  continuous  observations  are  straightforward. 

We  begin  by  briefly  reviewing  the  general  form  of  an  IRT 
model.  The  probability  of  response  x^  (1  if  correct,  0  if  not) 
from  person  i  to  item  j  is  given  by  an  IRT  model  as 

x. .  1-x. . 

p(x  |*if/J  )  =  [  f  ( 0 .  ,  /3  )  ]  lJ[l-f<*.,0  )]  1J  (1) 

where  6^  and  f)  are  real  (and  possibly  vector -valued)  parameters 

associated  with  person  i  and  item  j  respectively,  and  f  is  a 

known,  twice-differentiable ,  function  whose  range  is  the  unit 

interval.  Under  the  usual  IRT  assumption  of  local  independence, 

the  conditional  probability  of  the  response  pattern  - 

(x., . x.  )  of  person  i  to  n  items  is  the  product  of  n 

ll  in 


expressions  like  (1): 
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p(x.|0.,/9)  -  n  p(x  \8  VP.)  . 

j-1  J  J 


It  may  possible  to  express  item  parameters  as  functions  of 
some  smaller  number  of  more  basic  parameters  a  -  (a^, • • • ,“^)  that 
reflect  the  effects  of  M  salient  characteristics  of  items;  i.e., 

0  -  /L(a).  An  important  example  of  this  type  is  the  Linear 

Logistic  Test  Model  (LLTM;  Fischer,  1973,  Schieblechner ,  1972). 
Under  the  LLTM,  the  item  response  function  is  the  one -parameter 
logistic  (Rasch)  model,  or 


p[x_  |  0.  ,/9j  (a)  ]  -  exp[xj,j  (5.-^  )  ]/[l+exp(Si-/3j  )  ]  , 
and  the  model  for  item  parameters  is  linear: 


fi.(a)  =  2  Q.  a 

J  Jm  1,1 


The  elements  of  a  are  contributions  to  item  difficulty  associated 

with  the  M  characteristics  of  items,  presumably  related  to  the 

number  or  nature  of  processes  required  to  solve  them.  The 

elements  of  the  known  vector  Q.  indicate  the  extent  to  which  item 

J 

j  exhibits  each  characteristic.  Fischer  (1973),  for  example, 

models  the  difficulty  of  the  items  in  a  calculus  test  in  terms  of 

the  number  of  times  an  item  requires  the  application  of  each  of 

seven  differentiation  rules.  Q.  is  the  number  of  times  that  rule 

I™ 

m  must  be  employed  in  order  to  solve  Item  j . 
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Consider  now  a  set  of  items  that  may  be  answered  by  means  of 
K  different  strategies.  It  need  not  be  the  case  that  all  are 
equally  effective,  nor  even  that  all  generally  lead  to  correct 
responses.  Not  all  strategies  need  be  available  to  all 
examinees.  We  make  the  following  assumptions. 


1.  Each  examinee  is  applying  the  same  one  of  these  strategies 
for  all  the  items  in  the  set.  (In  the  final  section,  we 
discuss  prospects  for  relaxing  this  assumption  to  allow  for 
strategy- switching) . 

2.  The  responses  of  an  examinee  are  observed  but  the  strategy 
he  or  she  has  employed  is  not. 

3.  The  responses  of  examinees  following  Strategy  k  conform  to 
an  item  response  model  of  a  known  form. 

4.  Substantive  theory  posits  relationships  between  observable 
features  of  items  and  the  probabilities  of  success  enjoyed  by 
members  of  each  strategy  class.  The  relationships  may  be 
known  either  fully  or  only  partially  (as  when  the  Q  matrices 
in  LLTM-type  models  are  known  but  the  basic  parameters  are 
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Let  the  k'th  element  in  the  K-dimensional  vector  6.  take  the 

i 

value  one  if  examinee  i  follows  Strategy  k,  and  zero  if  not. 
Extending  the  notation  introduced  above,  we  may  write  the 
conditional  probability  of  response  pattern  as 


»<*ilW>  -  rn[fk<^V|Xi3il-fk<sik’V’  XlJ;<>Ik  <2) 

k  j  j  j 


where  /3  =/3.,  (a)  gives  the  item  parameter(s)  for  Item  j  under 

J  k  J  * 

Strategy  k. 

It  will  be  natural  in  certain  applications  to  partition  basic 

parameters  for  items  in  accordance  with  strategy  classes;  that  is, 

a  -  (a  When  there  are  K  versions  of  the  LLTM,  for 

i  K 

example,  differences  among  strategies  are  incorporated  into  the 

model  by  K  different  vectors  Q.,  ,  k—l,...,K,  that  relate  Item  j  to 

j  k 

each  of  the  strategies: 


^jk  ^  Qjkm  akm  Qjk  “k 


The  item  difficulty  parameter  for  Item  j  under  Strategy  k,  then, 
is  a  weighted  sum  of  elements  in  ,  the  basic  parameter  vector 
associated  with  Strategy  k;  the  weights  indicate  the  degree 

to  which  each  of  the  features  m,  as  relevent  under  Strategy  k,  are 
present  in  Item  j .  This  situation  will  be  illustrated  in  the 


following  example. 
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Example  1:  Alternative  strategies  for  spatial  tasks 
The  items  of  certain  tests  intended  to  measure  spatial 
visualization  abilities  admit  to  solution  by  nonspatial  analytic 
strategies  (French,  1965;  Kyllonen,  Lohman,  and  Snow,  1984; 
Pelligrino,  Mumaw,  and  Shute ,  1985).  Consider  items  in  which 
subjects  are  shown  a  drawing  of  a  three-dimensional  target 
object,  and  asked  whether  a  stimulus  drawing  could  be  the  same 
object  after  rotation  in  the  plane  of  the  picture.  In  addition 
to  rotation,  one  or  more  key  features  of  the  stimulus  may  differ 
from  the  those  of  target.  A  subject  may  solve  the  item  either  by 
rotating  the  target  mentally  the  required  degree  and  recognizing 
the  match  (Strategy  1) ,  or  by  employing  analytic  reasoning  to 
detect  feature  matches  without  performing  rotation  (Strategy  2) . 

Consider  further  a  hypothetical  three -item  test  comprised  of 
such  items.  Each  item  will  be  characterized  by  (1)  rotational 
displacement,  of  60,  120,  or  180  degrees,  and  by  (2)  the  number  of 
features  that  must  be  matched.  Table  1  gives  the  features  of  the 
items  in  the  hypothetical  test. 


Insert  Table  1  about  here 


Each  subject  i.  will  be  characterized  by  two  vectors.  In  the 
first,  4>  -  ^il’^i2^’  ^ik  ta^es  c^e  va^ue  ^  if  Subject  i  employs 
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Strategy  k  and  0  if  not.  In  the  second,  8 ^  -  (8 ^,8 »  0^ 
characterizes  the  proficiency  of  Subject  i  if  he  employs  Strategy 
k.  Only  one  of  the  elements  of  8 ^  is  involved  in  producing 
Subject  i's  responses,  but  we  do  not  know  which  one. 

Suppose  that  for  subjects  employing  a  rotational  strategy, 
probability  of  success  is  given  by  the  one -parameter  logistic 
(Rasch)  item  response  model: 

p(Xij l^il’^jl’^l"1)  "  exP[Xij(9il'^ji)l/t1+exP(5il‘^ji)]  • 


Here  0  is  the  proficiency  of  Subject  i  at  solving  tasks  by 


means  of  the  rotational  strategy,  and  0  is  the  difficulty  of 


Item  j  under  the  rotational  strategy. 

It  is  usually  found  that  the  time  required  to  solve  mental 
rotation  tasks  is  linearly  related  to  rotational  displacement.  To 
an  approximation,  so  are  log- odds  of  success  (Tapley  and  Bryden, 
1977).  We  assume  that  under  the  rotational  strategy,  item 
parameters  take  the  following  form: 


V  -  Qj 11  Q11  +  Q12  ’ 


where  Qj ^  encodes  the  rotational  displacement  of  Item  j--l  for  60 


degrees,  2  for  120  degrees,  and  3  for  180  degrees- -and  is  the 
incremental  increase  in  difficulty  for  each  increment  in  rotation; 
and  a^2  is  a  constant  term,  for  which  a  coefficient  is 

implied  for  all  items.  If  a..-l  and  a.  --2,  the  item  parameters 
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f)  ^  that  are  in  effect  under  Strategy  1  are  as  shown  in  the 
second  column  of  Table  2. 


Insert  Table  2  about  here 


A  Rasch  model  will  also  be  assumed  for  subjects  employing 
Strategy  2,  the  analytic  strategy,  but  here  the  item  parameters 
depend  on  the  number  of  features  that  must  be  matched: 


^ j 2  “  Qj21  “21  +  “22  ’ 


where  Qj2i  t*ie  number  °f  salient  features,  is  the 


incremental  contribution  to  item  difficulty  of  an  additional 
feature,  a ^  is  a  constant  term,  and  22""^  f°r  aH 

items.  If  a2l”^‘'’  and  a 2 2“* -2.5,  we  obtain  the  item  parameters 
that  are  in  effect  under  Strategy  2.  They  appear  in  the  third 
column  of  Table  2. 

Note  that  the  items  have  been  constructed  so  that  items  that 
are  relatively  hard  under  one  strategy  are  easy  under  the  other. 
Strategy  choice  cannot  be  inferred  from  observed  response  patterns 
unless  patterns  are  more  likely  under  some  strategies  and  less 
likely  under  others. 

The  response  pattern  Oil,  for  example,  has  a  correct  answer 


to  an  item  that  is  easy  under  the  Strategy  2  but  hard  under 
Strategy  1,  and  an  incorrect  answer  to  an  item  that  is  hard  under 
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Strategy  2  but  easy  under  Strategy  1.  Figure  1  plots  the 
likelihood  function  for  the  response  vector  Oil  under  both 
strategies;  that  is,  p[x=(011)  |  ^.^-l]  for  k-1 , 2  as  a  function  of 
0^  and  ^  respectively.  The  maximum  of  the  likelihood  under 
Strategy  2  is  about  eight  times  as  high  as  the  maximum  attained 
under  Strategy  1. 


Insert  Figure  1  about  here 


We  can  make  probabilistic  statements  about  individual 
subjects  if  we  know  the  proportions  of  people  who  choose  each 
strategy,  or  =  p(^^-=l)  ,  and  the  distributions  of  proficiency 
of  those  using  each  strategy  class,  or  =  p^k^k”^  ' 

Suppose  that  (i)  0^  and  8 ^  both  follow  standard  normal 
distributions  among  the  subjects  that  have  chosen  to  follow  them, 
and  (ii)  three  times  as  many  subjects  follow  Strategy  1  as  follow 
Strategy  2--i.e.,  =  3/4  and  =  1/4.  This  joint  prior 

distribution  is  pictured  in  Figure  2. 


£ 


Insert  Figure  2  about  here 


Routine  application  of  Bayes  theorem  then  yields  the  joint 
posterior  density  function  for  <i>  and  for  k-1 . K: 


k  6k 
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where 


Ptxl^k=1  .*  ./9k(a)  1  -  n  explx.j  (tf-/9^k(a)  ]  )/(  l+(5-/3jk(Q)  ]  J 


The  reciprocal  of  the  constant  of  proportionality  required  to 
normalize  (3)  is  the  marginalization  of  the  right  side,  or 


^  \  J  P(x|tf  L,  =  l  ,  8  <K  (Q>  1  66 


The  posterior  distribution  induced  by  (Oil)  is  shown  in  Figure  3. 
Marginalizing  with  respect  to  #k  amounts  to  summing  the  area  under 
the  curve  for  Strategy  k,  and  gives  the  posterior  probability  that 
4>k*=l--that  is,  that  the  subject  has  employed  Strategy  k.  The 
resulting  values  for  this  response  pattern  are  P(<^-1 |x-011)- . 28 
and  P(^2=l | x-011 )- . 72 .  The  prior  probabilities  favoring  Strategy 
1  have  been  revised  substantially  in  favor  of  Strategy  2.  The 
conditional  posterior  for  6^  given  <ji^-l  has  a  mean  and  standard 
deviation  of  about  .32  and  .80.  Corresponding  values  for  the 


distribution  of  6  given  are  -50  and  .81. 


Insert  Figure  3  about  here 


Parameter  Estimation 

This  section  discusses  estimation  procedures  for  mixtures  of 
IRT  models.  A  two-stage  procedure  is  described.  The  first  stage 
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integrates  over  6  and  <f>  distributions  to  obtain  a  so-called 
marginal  likelihood  function  for  the  structural  parameters  of  the 
problem- -the  basic  parameters  for  items,  the  proportions  of 
subjects  employing  each  strategy,  and  the  parameters  of  the  6 
distributions  of  subjects  employing  each  strategies.  Maximum 
likelihood  estimates  are  obtained  by  maximizing  this  likelihood 
function.  If  preferred,  Bayes  modal  estimates  can  be  obtained  by 
similar  numerical  procedures  by  multiplying  the  likelihood  by 
prior  distributions  for  the  structural  parameters.  The  second 
stage  takes  the  resulting  point  estimates  of  structural  parameters 
as  known,  and  calculates  aspects  of  the  posterior  distribution  of 
an  individual  examinee -- e  .  g .  ,  p((^^-l|x)  and  p(fi^|^^-l  ,x)  . 

Stage  1:  Estimates  of  Structural  Parameters 

Equation  2  gives  the  conditional  probability  of  the  response 
vector  x  given  6  and  <j> ,  or  p (x  |  0  ,  <£  ,at)  .  Consider  a  population  in 
which  strategies  are  employed  in  proportions  and  within- 
strategy  proficiencies  have  densities  Sj^^k^k^  among  the 
examinees  using  them.  The  marginal  probability  of  x  for  an 
examinee  selected  at  random  from  this  population  is 

p(x|o,  n,ri)  -  2  *k  f  p  (x  |  0k,*k-l  ,q)  gk(0kbk)  d#k  .  (4) 

k 

For  brevity,  let  £  denote  the  extended  vector  of  all  structural 
parameters,  namely  (a,n,rj).  The  logl ikelihood  for  £  induced  by 
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the  observation  of  the  response  vectors  X  -  (x, .... ,x.,)  of  N 

1  N 

subjects  is  a  constant  plus  the  sum  of  the  logs  of  terras  like  (4) 
for  each  subject: 

N 

A  =  2  log  p  ( x  .  |0 
i-1 

-  z  J  *ik  log  f  p[xi gk(*kK>  d*k 

i  k 

+  2  2  4>ik  log  xk  .  (5) 

i  k 


Let  S  be  the  vector  of  first  derivatives,  and  H  the  matrix  of 
second  derivatives,  of  A  with  respect  to  £.  Under  regularity 
conditions,  the  maximum  likelihood  estimates  £  solve  the 
likelihood  equation  S=0 ,  and  a  large-sample  approximation  of  the 
matrix  of  estimation  errors  is  given  by  the  negative  inverse  of  H 

A 

evaluated  at  £ . 

A  standard  numerical  approach  to  solving  likelihood 
equations  is  to  use  some  variation  of  Newton's  method.  Newton- 


Raphson  iterations,  for  example,  improve  a  provisional  estimate 
by  adding  the  correction  term  -H  ^  S  0.  Fletcher-Powell 
iterations  avoid  computing  and  inverting  H  by  using  an 
approximation  of  H  ^  that  is  built  up  from  changes  in  S  from  one 


cycle  to  the  next. 


These  solutions  have  the  advantage  of  rapid  convergence  if 
starting  values  are  reasonable  - -of ten  fewer  than  10  iterations 
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are  necessary.  S  and  H  can  be  difficult  to  work  out,  however, 
and  all  parameters  must  be  usually  be  dealt  with  simultaneously 
because  the  off-diagonal  elements  in  H  needn't  be  zero.  For  these 
reasons,  a  computationally  simpler  but  slower-converging  solution 
based  on  Dempster,  Laird,  and  Rubin's  (1977)  EM  algorithm  will  now 
be  described  as  well.  The  approximation  uses  discrete 


representations  for  the  g^s ,  so  the  relatively  simple  "finite 
mixtures"  case  obtains  (Dempster,  Laird,  and  Rubin,  1977) 

Suppose  that  for  each  k,  subject  proficiency  under  Strategy  k 

can  take  only  the  L(k)  values  8^ . ®kL(k)  '  dens^ty  %  is 

thus  characterized  by  these  points  of  support  and  by  the  weights 
associated  with  each,  I  *7^)  •  Define  the  subject  variable 


-  (V’ill . tf>iKL(K)>*  a  vector  of  length  5^  L(k) 


where  the  element 


t/>^ jj  i-s  1  if  the  proficiency  of  Subject  i  under  Strategy  k  is  8 
and  0  if  not.  There  are  a  total  of  K  Is  in  ip .  ,  one  for  each 

l 


ki 


strategy- -  though  again,  only  only  of  them  is  involved  in  producing 
x^--the  one  associated  with  the  strategy  that  Subject  i  happens  to 
employ.  Summations  replace  integrations  in  the  logl ikel ihood , 
which  can  nov  be  written  as 


A  =  s  ^ik  *  ^iki  los  P[xilfik-ek'V1-\(a) 

l  k  X 


+  2  ^ik  *  0i ki  6k(ek*lv 

i.  k  X 


+  2  Z  4>  .  .  log  7T. 


(6) 
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If  values  of  4>  and  t/>  were  observed  along  with  values  of  x,  ML 
estimation  of  £  from  (6)  would  be  simpler.  The  basic  parameter  a 
appears  only  in  the  first  term  on  the  right  side  of  (6) ,  so  that 
maximizing  with  respect  to  a  need  address  that  term  only.  When  a 
consists  of  distinct  subvectors  for  each  strategy,  even  these 
subvectors  lead  to  distinct  maximization  problems  of  lower  order. 
The  subpopulation  parameters  rj  appear  in  only  the  second  term, 
separating  them  in  ML  estimation;  they  too  lead  to  even  smaller 
separate  subproblems  if  77  consists  of  distinct  subvectors  for  each 
strategy.  The  population  proportions  *  appear  in  only  the  last 
term.  Unless  they  are  further  constrained,  their  ML  estimates  are 
simply  observed  proportions.  The  values  of  8  may  be  either 
specified  a  priori  (as  in  Mislevy,  1986)  or  estimated  from  the 
data  (as  in  de  Leeuw  and  Verhelst,  1986).  In  the  latter  case, 
their  likelihood  equations  have  contributions  from  both  the  first 
and  second  terms,  but  the  equations  for  the  points  of  support 
under  Strategy  k  involve  data  from  only  those  subjects  using 
Strategy  k.  Their  cross  second  derivatives  with  points 
corresponding  to  other  strategies  are  zero,  although  their  cross 
derivatives  with  elements  of  a  and  rj  that  are  involved  with  the 
same  strategy  need  not  be. 

The  M-step  of  an  EM  solution  requires  solving  a  maximization 
problem  of  exactly  this  type,  with  one  exception:  the  unobserved 
values  of  each  <j>  ^  and  0.  ate  replaced  by  their  conditional 
expections  given  and  provisional  estimates  of  £,  say  The 
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E-step  calculates  these  conditional  expectations  as  follows. 
Denote  by  1  ^  the  following  term  in  the  marginal  likelihood 
associated  with  Subject  i,  Strategy  k,  and  proficiency  value  9^ 
within  Strategy  k: 

xiki  -  p[xil\"eki’V1'4(a)]  Sk(eklV  *k  • 


The  required  conditional  expectations  are  obtained  as 


^iki  “  E(*ikflXi’^0) 


Xiki  7  ^,1iki' 


(7) 


and 


4  -  E<*lkivf-<°> 


(8) 


The  EM  formulation  makes  it  clear  how  each  subject 
contributes  to  the  estimation  of  the  parameters  in  all  strategy 
classes,  even  though  it  is  assumed  that  only  one  of  them  was 
relevant  to  the  production  of  his  responses.  His  data  contribute 
to  estimation  for  each  strategy  class  is  in  the  proportion  to  the 


*.  v  v  vv  v.v.v  v v  wwtv -  WAvra 
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probability  that  that  strategy  was  the  one  he  employed,  given  his 
observed  response  pattern. 

In  addition  to  its  simplicity,  the  EM  solution  has  the 
advantage  of  being  able  to  proceed  from  even  very  poor  starting 
values.  The  slowness  with  which  it  converges  can  be  a  serious 
drawback,  however.  Its  rate  of  convergence  depends  on  how  well  x 
determines  examinees'  9  and  <t>  values.  Accelerating  procedures 
such  as  those  described  by  Ramsay  (1975)  and  Louis  (1982)  can  be 
used  to  hasten  convergence. 


Stage  2:  Posteriors  for  Individual  Examinees 

When  the  population  parameters  £  are  accurately  estimated, 
the  posterior  density  of  the  parameters  of  examinee  i  is 
approximately 


p(5ik=«,^ik=1|xi.o  a  ptxJ^-M./y*)]  *k  gk(*hk)  . 


where  the  reciprocal  of  the  normalizing  constant  is  obtained  by 
first  integrating  the  expression  on  the  right  over  9  within  each 
k,  then  summing  over  k.  The  posterior  probability  that  Subject  i 
used  Strategy  k  is  approximated  by 


P(*ik'l|xi'°  “  J  p(*ik“*'*ik“l|xi'°  AS 


9 

81 
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The  examinee's  posterior  mean  and  variance  for  a  given  strategy 
class,  given  that  that  was  the  strategy  employed,  are  approximated 
by 

?ik  ’  f  '  p<Sik'#,*lk-1lTrS)  “»  / 

and 

If  the  discrete  approximation  has  been  employed,  (7)  and  (8) 
apply. 

Example  2:  A  Mixture  of  Valid  Responders  and  Random  Guessers 
Given  appropriate  instructions,  examinees  will  omit 
multiple-choice  test  items  when  they  don't  know  the  answers 
rather  than  guess  at  random.  The  Rasch  model  may  provide  a  good 
fit  to  such  data  if  omits  are  treated  as  incorrect.  If  a  small 
percentage  of  examinees  responds  at  random  to  all  items,  however, 
their  responses  will  bias  the  estimation  of  the  item  parameters 
that  pertain  to  the  majority  of  the  examinees. 

We  may  posit  a  two-class  model,  under  which  an  examinee 
responds  either  in  accordance  with  the  Rasch  model  or  guesses 
totally  at  random.  For  examinees  in  the  latter  class, 
probabilities  of  correct  response  are  constant,  e.g.,  at  the 
reciprocal  of  the  number  of  response  alternatives  to  each  item. 
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Using  the  procedures  described  in  the  preceding  sections,  it  is 
possible  to  free  estimates  of  the  item  parameters  that  pertain  to 


the  valid  responders  from  biases  due  to  random  guessers,  even 
though  it  is  not  known  with  certainty  who  the  guessers  are. 

A  mixture  model  for  the  (marginal)  probability  of  response 


pattern  x  in  this  situation  is 


P(x.|0  =  £  P(x.|4k=l,0  *k  , 
k=l 


where  Strategy  Class  1  is  the  Rasch  model  and  Class  2  is  random 


guessing.  The  composition  of  £  is  now  described.  It  includes 
first  the  strategy  proportions  and  .  For  the  Rasch  class,  the 
basic  parameters  are  item  difficulty  parameters  b^  for 
j=l,...,n.  Suppose  the  distribution  g^  of  proficiencies  of 
subjects  following  the  Rasch  model  is  discrete,  with  L  points  of 

support  0  -  (0^ . 0^)  and  associated  weights  w  =  (u^, . . . ,u^) . 

The  (marginal)  probability  of  response  pattern  x  under  Strategy  1 


P(xk=l,o  ,P,u)  -  X  L>  n  exp[x .  (9  -b  )  ]/[  l+exp(9  -b  ) 

£  j  J  J  J 


Under  the  random  guessing  strategy,  the  basic  parameters  a ^  are 
the  probabilities  c^  of  responding  correctly  to  each  item  j.  All 
subjects  following  this  strategy  are  assumed  to  have  the  same 
probabilities  of  correct  response,  so  no  distribution  g^  enters 


n*  a  "«»  v  v  v  v  v.v  v,v  ^^r^jVJW,JWVcwwTCW7WirjinnrnrLTr.'w*iTr,iwir„-ir, 
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the  picture.  For  such  subjects,  the  probability  of  response 
pattern  x  is  simply 


x .  1 -x . 

P(x|tf2-l,a2)  -  n  c  J  (1-c  )  J  . 


An  artificial  dataset  was  created  for  four  items  under  this 
model  in  accordance  with  the  following  specifications.  Of  1200 
simulees  in  all,  1000  followed  the  Rasch  model  and  200  were  random 
guessers,  implying  *^=.833  and  jr  -.167.  The  Rasch  item 
parameters  were  =  (b^,...,b^)  =  ( - . 511 , - . 105 , . 182 , . 403) .  A 
discrete  density  with  six  points  of  support  was  used  to  create  the 
data  for  the  Rasch  class.  The  points  and  their  corresponding 
proportions  were  as  follows: 


Point  Proportion 


■1.204 
-  .357 
.095 
.262 
.470 
.642 


The  rates  of  correct  response  for  the  random  guessers  on  the  four 


items  were  a?  =  (c^ 


,c,  )  =  (  .  30,  .35,  .20,  .  15)  .  The 
4 


probability  of  each  of  the  sixteen  possible  response  patterns  was 
calculated  within  each  class,  multiplied  by  the  number  of  simulees 
in  that  class,  summed  over  classes,  and  rounded  to  the  nearest 


integer.  The  resulting  data  are  shown  in  Table  3. 
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Insert  TabLe  3  about  here 


A  standard  Rasch  model  was  first  fit  to  the  data  using  the 
two-step  marginal  maximum  likelihood  procedures  described  by  de 
Leeuw  and  Verhelst  (1986).  Conditional  maximum  likelihood  (CML) 
estimates  were  first  obtained  for  item  parameters.  Setting  their 
scale  by  centering  them  around  zero  like  the  true  item  parameters 
for  the  Rasch  class,  the  resulting  values  were  (-.329,  -.053, 

.127,  .252).  Note  that  these  values  are  biased  toward  their 
center;  the  presence  of  random  guessers  blurs  the  distinctions 
among  the  differences  in  item  difficulties.  A  three-point 
discrete  distribution- -  the  greatest  number  of  points  leading  to  an 
identified  model  for  a  four-item  test- -was  next  estimated  for 
subjects.  The  expected  counts  of  response  patterns  under  this 
model  are  also  shown  in  Table  3.  A  chi-square  of  7.16  with  8 
degrees  of  freedom  results,  indicating  an  acceptable  fit  for  a 
sample  of  the  size  we  have  employed. 

A  mixture  model  of  the  generating  form  was  then  fit  to  the 
data,  with  two  exceptions.  First,  the  multiplicative  form  of  the 
Rasch  model  was  employed  during  calculations.  Since  maximum 
likelihood  estimates  are  invariant  under  transformations,  the 
estimates  of  the  structural  parameters  obtained  under  the 
multiplicative  form  need  merely  be  transformed  back  to  the  usual 


»,y 

•Jy 

& 
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additive  form  shown  above.  Second,  a  three-point  discrete 
distribution  was  again  employed  for  the  Rasch  class,  with  the 
lowest  point  fixed  at  zero  in  the  multiplicative  scale.  This 
corresponds  to  0^  =  -«  in  the  additive  scale,  implying  incorrect 
responses  to  all  items  with  probability  one.  (As  it  turns  out, 
the  estimated  weight  associated  with  this  point  will  be  zero.) 
The  total  number  of  parameters  to  be  estimated,  then,  was  13: 


2  free  points  in  the  Rasch  distribution:  0^  and  0^. 

2  free  values  for  weights  at  the  three  points  in  the  Rasch 

distribution:  ,  w  and  ,  where  S  -  1 . 

4  item  parameters  for  the  Rasch  class:  a. - (b. , . . . , b, ) . 

11  U 

4  item  parameters  for  the  guessing  class:  . c^) . 

1  relative  proportion  for  class  representation:  . 


In  light  of  the  fact  that  only  13  degrees  of  freedom  are 
available  from  the  data,  in  the  form  of  16  response  patterns  whose 
counts  that  must  sum  to  1200,  an  unaccelerated  EM  solution 
converged  painfully  slowly.  FI etche r - Powe 1 1  iterations  were 
emploved  instead,  and  they  converged  rapidlv.  The  Rasch-onlv 
estimates  described  above  were  used  as  starting  values  for  the 
Rasch  class  item  parameters  and  population  distribution.  For  the 
c's,  a  common  value  midwav  among  the  true  values  was  used.  For 
,  starting  values  of  .10,  .13,  and  .20  were  used  in  three 

different  runs.  All  runs  converged  to  the  same  solution: 


. U*.  I.- . to*  . H. 


=  (-. 501, - .091, .193, .398) ; 

=  (-«,  -.534,  .354); 

=  (<10" 10 ,  .  319,  .681) ; 

=  (.287,  .230,  .179,  .139); 
=  .164. 
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Although  the  c's  are  slightly  underestimated,  the  structure  of 


the  data  has  been  reconstructed  quite  well.  The  expected  counts 
of  response  patterns  are  also  shown  in  Table  3.  As  they  should, 
they  yield  a  nearly  perfect  fit:  a  chi-square  of  .008  on  3  degrees 
of  freedom.  The  improvement  in  chi-square  is  dramatic  if  not 
significant- - it  would  be  for  larger  samples  or  longer  tests--but 
the  removal  of  the  bias  in  the  Rasch  item  parameter  estimates  is 


the  point  of  the  exercise. 

Table  4  shows  conditional  likelihoods  of  each  response 
pattern  given  that  an  examinee  is  a  guesser,  a  member  of  the 
Rasch  class  with  #=-.534,  and  a  member  of  the  Rasch  class  with 
#=.354.  The  estimated  proportions  of  the  population  in  these 
categories  are  .164,  .267,  and  .569  respectively.  Multiplying 


these  population  probabilities  times  a  pattern's  corresponding 
likelihood  terms,  then  normalizing,  gives  the  posterior 
probabilities  that  also  appear  in  the  table.  Posterior 
probabilities  are  given  for  membership  in  the  guessing  class,  and 
for  #=-.534  and  #=.354  given  membership  in  the  Rasch  class. 
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Insert  Table  4  about  here 


Recall  from  the  description  of  the  EM  solution  that  the  data 
from  an  examinee  is  effectively  distributed  among  strategy 
classes  to  estimate  the  item  parameters  within  that  class.  This 
means  that  the  responses  of  all  examinees  play  a  role  in  both 
estimating  both  b's  and  c's--but  with  weights  in  proportion  to  the 
posterior  probabilities  shown  in  Table  4.  From  responses  to  only 
four  items,  we  never  have  overwhelming  evidence  that  a  particular 
examinee  is  a  guesser.  Only  those  with  all  incorrect  responses 
can  be  judged  more  likely  than  not  to  have  guessed.  Had  only 
those  respondents  been  treated  as  guessers - -  and  that  would  be  the 
Bayesian  modal  estimate  of  strategy  class- -estimated  c's  would  all 
have  been  zero.  But  employing  a  proportion  of  data  from  all 
patterns,  even  those  with  all  items  correct,  yields  estimated  c's 
that  essentially  recover  the  generating  values. 

As  a  consequence  of  using  the  Rasch  model  for  Strategy  1, 
the  conditional  posterior  distributions  given  that  a  subject 
belongs  to  this  class,  or  p(0|x,<^=l),  are  identical  for  all 
response  patterns  x  with  the  same  total  score.  The  probability 
that  an  examinee  belongs  to  the  Rasch  class  vary  considerably 
within  patterns  with  the  same  score,  however.  For  any  given 


response  pattern,  the  posterior  probability  of  being  in  the  Rasch 
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class  can  be  inferred  from  Table  4  as  1  -  For  patterns 

with  exactly  one  correct  response,  these  probabilities  are,  for 
Items  1-4  in  turn,  .869,  .800,  .687,  and  .519. 

Discussion 

Theories  about  the  processes  by  which  examinees  attempt  to 
solve  test  items  play  no  role  in  standard  applications  of  test 
theory,  including  conventional  item  response  theory.  Only  a  data 
matrix  of  correct  and  incorrect  responses  is  addressed,  and  items 
and  examinees  are  parameterized  strictly  on  the  basis  of 
propensities  toward  correct  response.  When  all  that  is  desired  is 
a  simple  comparison  of  examinees  in  terms  of  a  general  propensity 
of  this  nature,  IRT  models  suffice  and  in  fact  offer  many 
advantages  over  classical  true-score  test  theory. 

Situations  for  which  standard  IRT  models  prove  less 
satisfactory  involve  a  desire  either  to  better  understand  the 
cognitive  processes  that  underlie  item  response,  or  to  employ 
theories  about  such  processes  to  provide  more  precise  or  more 
valid  measurement.  Extensions  of  item  response  theory  in  this 
direction  are  exemplified  by  the  Linear  Logistic  Test  Model 
(Schieblechner ,  1972;  Fischer,  1973),  Embretson's  (1985) 
multicomponent  models,  Samejima'a  (1983)  model  for  multiple 
strategies,  and  Tatsuoka's  (1983)  "rule  space”  analyses. 

The  approach  offered  in  this  paper  concerns  situations  in 
which  different  persons  may  choose  different  strategies  from  a 
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number  of  known  alternatives,  but  overall  proficiencies  provide 
meaningful  comparisons  among  persons  employing  the  same  strategy. 
We  suppose  that  strategy  choice  is  not  directly  observed  but  can 
be  inferred  (with  uncertainty)  from  response  patterns  on 
theoretical  bases.  Assuming  that  substantive  theory  allow  us  to 
differentiate  our  expectations  about  response  patterns  under 
different  strategies,  and  that  a  subject  applies  the  same  strategy 
on  all  items,  it  is  possible  to  estimate  the  parameters  of  IRT 
models  for  each  strategy.  It  is  further  possible  to  calculate  the 
probabilities  that  a  given  subject  has  employed  each  of  the 
alternative  strategies,  and  estimate  his  proficiency  under  each 
given  that  that  was  the  one  he  used. 

Assuming  that  a  subject  uses  the  same  strategy  on  all  items 
is  obviously  undesirable  for  manv  important  problems.  In  a 
technical  sense,  the  approach  can  be  extended  to  allow  for 


strategy- sw 

itching  bv 

defining  addi 

tional  strategy 

c  1 

asses 

that 

are 

in  e  f  f e 

ct.  combi  nut: 

ions  of  diffo 

rent 

st  rat 

:egies 

fo 

r  different 

i  te 

ms.  Bas 

od  on  Just 

and  Carpenter 

'  s  ( 

198b) 

f  i  ndi 

n& 

that 

sub j  net s 

sometimes  a 

pplv  whicho 

vor  strategy 

is  e 

a  s  i  e  r 

for  a 

von  p 

roblem , 

we 

might  do 

fine  throe 

strategy  clas 

ses 

for  it 

:ems  1 

ike 

thos 

e  in  our 

Examp  1 e  1  : 

o 

Always 

apply  the 

rotational  st 

rate 

p.v; 

o 

Always 

apply  the 

analytic  stra 

t  egv 

o 

Appl  v 

whichever  s 

; tratogv  is  he 

t  ter 

suit* 

•d  to 

an 

i  t  cm 

Different  Strategies 

28 


If  items  were  constructed  to  run  from  easy  to  hard  under  the 
rotational  strategy  and  hard  to  easy  under  the  analytic,  subjects 
using  the  third  "mixed"  strategy  would  find  them  easy,  then 
harder,  then  easier  again. 

There  are  limitations  to  how  far  these  ideas  can  be  pressed 
in  applications  with  binary  data.  Our  second  example  showed  that 
the  misspecified  Rasch  model  fit  a  four-item  test  acceptably  well 
with  a  sample  of  1200  subjects;  in  one  way  or  another,  more 
information  would  be  needed  to  attain  a  sharper  distinction 
between  strategy  classes  and,  correspondingly,  more  power  to 
differentiate  among  competing  models  for  the  data.  One  source  of 
information  is  more  binary  items.  Fifty  items  rather  than  four, 
including  some  that  are  very  hard  under  the  Rasch  strategy,  would 
do.  A  different  source  of  information  available  in  other  settings 
would  be  to  draw  from  richer  observational  possibilities. 

Examples  would  include  response  latencies  as  well  as  correctness, 
eye-fixation  patterns,  and  choices  of  incorrect  alternatives  that 
are  differentially  likely  under  different  strategies. 

Differentiating  the  likelihood  of  response  patterns  under 
different  strategies  is  the  key  to  successful  applications  of  the 
approach.  Its  use  would  be  recommended  when  identifying  strategy 
classes  is  of  primary  importance  to  the  selection  or  placement 
decision  that  must  be  made,  and  overall  proficiency  is  of 
secondary  importance.  The  items  in  the  test  must  then  be 
constructed  to  maximize  strategy  differences,  e.g.,  using  items 
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that  are  hard  under  one  strategy  but  easy  under  another.  Most 
tests  in  current  use  with  standard  test  theory  are  not  constructed 
with  this  purpose  in  mind;  indeed,  they  are  constructed  so  as  to 
minimize  differentiation  among  strategies,  since  it  lowers  the 
reliability  of  overall-propensity  scores.  When  strategy  class 
decisions  are  of  interest,  a  conventional  tests  is  not  likely  to 
provide  useful  information.  (Although  a  battery  of  conventional 
tests  might;  differences  in  score  profiles  are  analogous  to 
differential  likelihoods  of  item  response  patterns,  but  at  a 
higher  level  of  aggregation.) 

In  addition  to  the  applications  used  in  the  preceding 
examples,  a  number  of  other  current  topics  in  educational  and 
psychological  research  are  amenable  to  expression  in  terms  of 
mixtures  of  IRT  models.  We  conclude  by  mentioning  three. 

Hierarchical  development.  Wilson’s  (1984,  1985)  "saltus" 
model  (Latin  for  "leap")  extends  the  Rasch  model  to  developmental 
patterns  in  which  capabilities  increase  in  discrete  stages,  by 
including  stage  parameters  as  well  as  abilities  for  persons,  and 
stage  parameters  as  well  as  difficulties  for  items.  Examples 
would  include  Piaget's  (I960)  innate  developmental  stages  and 
Gagne's  (1962)  learned  acquisition  of  rules.  Suppose  that  K 
stages  are  ordered  in  ti rms  of  increasing  and  cumulative 
competence.  In  our  notation,  <$>  would  indicate  the  stage 
membership  of  a  subject  .  in  the  highest  stage,  item  responses 
follow  a  Rasch  model  with  parameters  h. .  Rasch  models  fit  lower 
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stages  as  well,  but  the  item  parameters  are  offset  by  amounts  that 
depend  on  which  stage  the  item  can  first  be  solved.  Our  basic 
parameters  a  would  correspond  to  the  item  parameters  for  the 
highest  stage  and  the  offset  parameters  for  particular  item  types 
at  particular  lower  stages.  Figure  4  gives  a  simple  illustration 
in  which  items  associated  with  higher  stages  have  an  additional 
increment  of  difficulty  for  subjects  at  lower  stages.  In 
applications  such  as  Siegler's  (1981)  balance  beam  tasks,  subjects 
at  selected  lower  stages  tend  to  answer  certain  types  of  higher- 
stage  items  correctly  for  the  wrong  reasons.  In  these  cases,  the 
offset  works  to  give  easier  item  difficulty  parameters  to  those 
items  in  those  stages. 


Insert  Figure  4  about  here 


Mental  models  for  problem  solving.  In  the  introduction  to 
their  experimental  study  on  mental  models  for  electricity, 
Centner  and  Gentner  (1983)  state 


Analogical  comparisons  with  simple  or  familiar  systems  often 
occur  in  people's  descriptions  of  complex  systems,  sometimes 
as  explicit  analogical  models,  and  sometimes  as  implicit 
analogies,  in  which  the  person  seems  to  borrow  structure  from 
the  base  domain  without  knowing  it.  Phrases  like  "current 
being  routed  along  a  conductor"  and  "stopping  the  flow"  of 
electricity  are  examples  (p.  99). 


Mental  models  are  important  as  a  pedagogical  device  and  as  a 
guide  to  problem-solving.  Inferring  which  models  a  person  is 
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using,  based  on  a  knowledge  of  how  conceivable  analogues  help  or 
hinder  the  solution  of  certain  types  of  problems,  provides  a 
guide  to  subsequent  training.  In  Gentner  and  Gentner's 
experiment,  the  problems  concerned  simple  electrical  circuits  with 
series  and  parallel  combinations  of  resistors  and  batteries. 
Popular  analogies  for  electricity  are  flowing  waters  (Strategy  1) 
and  "teeming  crowds"  of  people  entering  a  stadium  through  a  few 
narrow  turnstiles  (Strategy  2) .  The  water  flow  analogy 
facilitates  battery  problems,  but  does  not  help  with  resistor 
problems;  indeed,  it  suggests  an  incorrect  solution  for  the 
current  in  circuits  with  parallel  resistors.  The  teeming  crowd 
analogy  facilitates  problems  on  the  combination  of  resistors,  but 
is  not  informative  about  combinations  of  batteries.  If  a  Rasch 
model  holds  for  items  within  strategies,  Gentner  and  Gentner's 
hypotheses  correspond  to  constraints  on  the  order  of  item 
difficulties  with  the  two  strategies.  If  each  item  type  were 
replicatated  enough  times,  it  would  be  possible  to  make  inferences 
about  which  model  a  particular  examinee  was  using,  in  order  to 
plan  subsequent  instruction. 

Changes  in  intelligence  over  age.  An  important  topic  in  the 
field  of  human  development  is  whether,  and  how,  intelligence 
changes  as  people  age  (Birren,  Cunningham,  and  Yamamoto,  1983). 
Macrae  (n.d.)  identifies  a  weakness  of  most  studies  that  employ 


psychometric  tests  to  measure  aging  effects:  total  scores  fail  to 
reflect  important  differences  in  the  strategies  different  subjects 
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bring  to  bear  on  the  items  they  are  presented.  Total  score 
differences  among  age  and  educational-background  groups  on  Raven's 
matrices  test  were  not  significant  in  the  study  she  reports.  But 
analyses  of  subjects'  introspective  reports  on  how  they  solved 
items  revealed  that  those  with  academically  oriented  background 
were  much  more  likely  to  have  used  the  preferred  "algorithmic" 
strategy  over  a  "holistic"  strategy  than  those  with  vocationally 
oriented  backgrounds.  Since  the  use  of  algorithmic  strategies  was 
found  to  increase  probabilities  of  success  differentially  on 
distinct  item  types,  this  study  would  be  amenable  to  IRT  mixture 
modeling.  Inferences  could  then  be  drawn  about  problem-solving 
approaches  without  resorting  to  more  expensive  and  possibly 
unreliable  introspective  evidence. 
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Item  Features 


Item 

rotational 

displacement 

salient  features 

1 

60 

degrees 

3 

2 

120 

degrees 

2 

3 

180 

degrees 

1 

Item 


Table 

Item  Difficulty 


2 

Parame  ter s 


Strategy  1 


Strategy  2 


1 

-1 

.0 

2 

.0 

2 

0. 

.0 

0. 

.5 

3 

1 

.0 

-1 

.0 

Table  4 


Response  Pattern  Likelihoods  and  Posterior  Probabilities 
x  l(x|<^2)  L(x|e2,^1)  L(x|e2,^1)  p (^>2 1 x)  P(e2|x,^1)  P(e3|x,^1) 


0000 

.388 

.  150 

.027 

.  534 

.  719 

.281 

0001 

.063 

.  131 

.058 

.131 

.513 

.487 

0010 

.085 

.107 

.047 

.200 

.513 

.487 

0011 

.014 

.093 

.  100 

.027 

.  303 

.697 

0100 

.116 

.080 

.036 

.  313 

.513 

.487 

0101 

.019 

.070 

.076 

.  047 

.  303 

.697 

0110 

.025 

.057 

.062 

.076 

.  303 

.697 

0111 

.004 

.050 

.131 

.008 

.151 

.849 

1000 

.156 

.053 

.024 

.481 

.513 

.487 

1001 

.025 

.047 

.050 

.092 

.303 

.697 

1010 

.034 

.038 

.041 

.143 

.  303 

.697 

1011 

.005 

.033 

.087 

.015 

.151 

.849 

1100 

.047 

.029 

.031 

.234 

.  303 

.697 

1101 

.008 

.025 

.065 

.027 

.  151 

.849 

1110 

.010 

.020 

.053 

.045 

.151 

.849 

1111 

.002 

.018 

.113 

.004 

.068 

.932 

Note:  denotes  membership 

4>2  denotes  membership 
0_  denotes  membership 
with  8=- . 534 ; 

0.,  denotes  membership 
with  9= . 354 . 


in  the  class  of  Rasch  responders; 
in  the  class  of  random  guessers; 
in  the  class  of  Rasch  responders, 

in  the  class  of  Rasch  responders 
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Figure  4 

Saltus  example:  3  stages,  common  offset 
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Smithsonian  Institution 
801  North  Pitt  Street 
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Dr.  R i chard  E.  Snow 
Department  of  Psychology 
Stanford  University 
Stanford,  CA  94306 

Dr.  Richard  Sorensen 
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Dr.  Paul  Speckman 
University  of  Missouri 
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Dr.  Judy  Spray 
ACT 
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200  North  Beauregard  Street 
A  I  exandr i a,  VA  2231 1 

Dr .  William  Stout 
University  of  Illinois 
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101  I  I  I  i n i  Hall 
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Washington,  DC  20330 

Dr.  Hariharan  Swaminathan 
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School  of  Education 
University  of  Massachusetts 
Amherst,  MA  01003 

Mr.  Brad  Sympson 

Navy  Personnel  R&D  Center 

San  Diego,  CA  92152-6800 
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Dr.  John  Tangney 
AFOSR/NL 

Bolling  APB,  DC  20332 
Dr.  Kikumi  latsuoka 

CERl 

252  Engineering  Research 
Laboratory 
Urbana,  IL  61801 

Dr.  Maurice  latsuoka 
220  Lducat ion  bldg 
1310  8.  Sixth  St. 

Champaign,  1L  61820 

Dr.  David  fhissen 
Department  of  f'sychology 
University  of  Kansas 
Laurence,  KS  66044 

Mr.  Gary  Thomasson 
University  of  Illinois 
Educational  Psychology 
Champaiqn,  IL  61820 

Dr.  Robert  fsutakawa 
University  of  Missouri 
Department  of  Statistics 
222  Math.  Sciences  Bldg. 
Columbia,  MO  652 1 1 

Dr.  Ledyard  lucker 
University  of  Illinois 
Department  of  Psychology 
603  E.  Daniel  Street 
Champaign,  IL  61820 

Dr.  Vern  W.  Urry 
Personnel  R&l)  Center 
Office  of  Personnel  Management 
1900  E.  Street,  NW 
Washington,  DC  20415 

Dr .  Dav id  Vale 
Assessment  Systems  Lorp. 

2233  University  Avenue 
Suite  310 

St.  Paul,  MN  551 14 

Dr.  frank  Vicino 

Navv  Personnel  R&D  Center 

San  Diego,  CA  92152-6800 


Dr .  Howard  Wa i ner 

Division  of  Psychological  Studies 
Educational  Testing  Service 
Princeton,  NJ  08541 

Ur.  Mi ng  Me i  Wang 
L i ndqu i st  Center 
for  Measurement 
University  of  Iowa 
Iowa  City,  1A  52242 

Dr .  Thomas  A .  Warm 
Coast  Guard  Institute 
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Oklahoma  City,  OK  73169 

Dr.  Brian  Waters 
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HumRRO 

1100  S.  Washington  St. 

A  I exandr i a,  VA  22314 

Dr .  Dav id  J .  Weiss 
N660  Elliott  Hall 
University  of  Minnesota 
/5  E .  River  Road 
Minneapolis,  MN  55455 

Ur.  Ronald  A.  We i tzman 
NPS,  Code  54Wz 
Monterey,  CA  92152-6800 

Major  John  Welsh 
AT  URL /MOAN 

Brooks  AT  B ,  TX  78223 

Dr .  Douglas  Wetzel 
Code  12 

Navy  Personnel  R&D  Center 
San  Diego,  CA  92152-6800 

Dr.  Rand  R.  Wilcox 
University  of  Southern 
Cal  i f  orn i a 

Department  of  Psychology 
Los  Angeles,  CA  90007 
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German  Military  Representative 
ATTN:  Wolfgang  Wildegrube 
Strei tkraefteamt 
D-5300  Bonn  2 

4000  Brandywine  Street,  NW 
Washington,  DC  20016 

Dr.  Bruce  Willi ams 
Department  of  Educational 
Psychology 

University  of  Illinois 
Urbana,  IL  61801 

Dr .  Hilda  Wing 
NRC  GF- 176 

2101  Constitution  Ave 
Washington,  DC  20418 

Dr.  Martin  F.  Wiskoff 
Navy  Personnel  R  &  D  Center 
San  Diego,  CA  92152-6800 

Mr .  John  H.  Wolfe 

Navy  Personnel  R&D  Center 

San  Diego,  CA  92152-6800 

Dr.  George  Wong 
B i ostat i st i cs  Laboratory 
Memorial  S I oan-Kette r i ng 
Cancer  Center 
1275  York  Avenue 
New  York,  NY  10021 

Dr.  Wallace  Wulfeck,  III 
Navy  Personnel  R&D  Center 
San  Diego,  CA  92152-6800 


Dr.  Anthony  R.  Zara 
National  Council  of  State 
Boards  of  Nursing,  Inc. 
625  North  Michigan  Ave. 
Suite  1 544 
Chicago,  IL  60611 


Dr.  Kentaro  Yamamoto 
Computer-based  Education 
Research  Laboratory 
University  of  Illinois 
Urbana,  IL  61801 

Dr.  Wendy  Yen 
CTB/McGraw  Hill 
Del  Monte  Research  Park 
Monterey,  CA  93940 

Dr.  Joseph  L.  Young 
Memory  &  Cognitive 
Processes 

National  Science  foundation 
Washington,  DC  20550 
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